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ABSTRACT 

Results of cognitive diagnoses were validated using 
the rule-space model, and the usefulness of cognitive diagnosis for 
instruction was demonstrated. Through preliminary studies, 33 stages 
of knowledge were defined as cognitive diagnoses to be applied to 
results from a test of knowledge about fractions of seventh and 
eighth graders (initial test groups of 287 and 191). Results strongly 
indicate that the rule-space model can effectively diagnose students' 
knowledge states and can point out ways for remediating their errors 
quickly with minimum effort. It was also found that the designing of 
instructional units for remediation can be effectively guided by the 
rule-space model, because the determination of all possible ideal 
item-score patterns, given an incidence matrix, is based on a tree 
structure of cognitive attributes, knowledge states, and items. 
Fourteen tables and six figures present study findings. Appendix 1 
contains the fraction addition problems, and Appendixes 2, 3, and A 
contain supplemental information about the analysis. (Author/SLD) 
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A Psychometrically Sound Cognitive Diagnostic Model: 
Effect of Remediation as Empirical Validity 

Abstract 

The purpose of this study was to validate the results of cognitive 
diagnoses using the rule-space model and to demonstrate the usefulness 
of cognitive diagnoses for instruction. The results of the study 
strongly indicated that the rule -space model can effectivaly diagnose 
students' knowledge states and can point out ways for remediating their 
errors quickly with minimiim effort. 

It was also found that the designing of instructional units for 
remediation can be effectively guided by the rule'-space model, because 
the determination of all possible ideal item-score patterns, given an 
incidence matrix, is based on a tree structure of cognitive attributes, 
knowledge states , and items . 



Introduction 

The diagnosing of cognitive errors committed by students taking a test 
is a matter of interest not only to teachers, but also to cognitive 
psychologists and scientists who investigate the cognitive processes that 
underlie problem solving and reasoning (Greeno & Simon, 1984) . To carry out 
such diagnoses adequately, it is necessary first to do a task analysis of the 
test by constructing what is called an incidence matrix, which associates with 
each item an attribute vector. The latter is a binary vector with Is and Os 
as elements, representing the presence or absence, respectively, of various 
basic-skill attributes in each item. That is, if the jth basic skill on a 
list is required for correctly answering a given item, then the jth element of 
the associated attribute vector is 1; otherwise it is 0. 

The determination of classification groups must be done prior to 
formulating a classification space, rule space which is defined in the next 
section. Tatsuoka (1991) and Varadi and Tatsuoka (198?) successfully 
introduced a Boolean algebra algorithm by which all possible knowledge states 
can be derived from the information embedded in the test items --more 
precisely, from an incidence matrix. The number of knowledge states can often 
be as large as one thousand. 

However, knowledge and capability states are impossible to observe 
directly unless we use computers for testing and prepare special computer 
programs such as intelligent tutoring systems. However, developing such 
intelligent systems is very expensive because they are essentially domain- 
specific and require extensive programming efforts. 

Since only item scores are observable in practical testing situations, 
one must develop a probabilistic method by which unobservable performances on 
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unobservable cognitive tasks can be inferred from .observable item scores . 

One of the assumptions used in this study is that only item scores are 
observable and the underlying cognitive tasks are not expected to be 
observable . 

The purposes of this study are: 1) To validate empirically whether or 
not the fraction diagnostic test based on the rule -space model is effectively 
classifying each examinee into his/her true state, and 2) to test whether 
diagnostic information is useful and efficient for remediation. 

A brief summary of ru^e-space model, a psychometrically sound cognitive 
diagnostic model, will be given in the next section, and our procedure for the 
empirical validation of the model will be introduced. Finally the results 
from our study will be discussed. 
Classification Space: Rule Space 

A convenient tool for facilitating error diagnosis is rule space, which 
was developed by Tatsuoka and her associates (Tatsuoka, 1983, 1985; Tatsuoka & 
Tatsuoka, 1987; Tatsuoka & Tatsuoka, 1989). One of the upshots of utilizing 
the rule -space model is that various "misconception groups" are representable 
by ellipses centered on what might be called "pure rule points"--i.e. , points 
onto which are mapped the response patterns resulting from the consistent use 
of specific erroneous rules of operation throughout the entire test (Tatsuoka, 
1986). 

The formulation of rule space is done by utilizing Item Response Theory 
(IRT) in order to maintain continuity with current psychometric theories. 
Item respons? functions are used to derive an index f (defined in the next 
section, Eq.7) that is sensitive to the unusualness of item response patterns 
(Tatsuoka, 1984; latsuoka & Linn, 1983). A Cartesian product space of IRT 

2 
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ability $ and the variable f is used to formulate a classification space. 

We first define a function f(x) that is proportional to the coefficient 
of regression of P(^)-x on P(d)-T(^), so that when its value is close to zero, 
it means that the two probability vectors are almost collinear. Since Pj(^) 
is the expectation of item score Xj given 0y the conditional expectation of 
f(x) equals zero. Therefore, when the elements of an observed vector x are 
close to the average performances on the test items, the absolute value of 6 
values will be nearly zero. 

Students' misconceptions diagnosed by the rule- space model can be 
related with the IRT ability, 0. The unusualness of diagnosed cognitive 
errors can be judged by examining -values because the expectation of ^ is 
zero. f -values close to zero, that is, close to the ^-axis, indicate that 
such item response patterns are frequently observed, which in turn means the 
corresponding knowledge states are observed for many students. If the J* -value 
associated with a knowledge state is large, then such a state is unusual. The 
larger a f -value is, the more unusual is the state associated with this value. 
A similar argument holds for negative f values . 

By locating the position of a knowledge state of interest in rule space, 
one can make two inferences: (1) What IRT-ability level is likely to produce 
this particular state, and (2) what percentage of students in a sample will be 
classified into this state. If some sources of error (or the lack of specific 
knowledge) are shared by many students, then the states involving such errors 
will be found closer to S-axis (Tatsuoka, 1990). 

Bug distribution The performance on the test items is not always 
perfectly consistent with the erroneous rule or "bug" (denoted by R) used most 
frequently by a student, and the responses that deviate from the modal rule 



are called slips or random errors. The item response patterns deviating by 
various numbers of slips from a pure rule pattern R will form a cluster around 
the pure rule pattern. We assume that occurrences of slips on items are 
independent across the items. Tatsuoka and Tatsuoka (1987) showed that the 
distribution of the number of slips is a compound binomial distribution if the 
slippage probabilities of the items are different, and a binomial distribution 
if the slippage probabilities are the same across items. 

Let us map all the "fuzzy" response patterns in the cluster around R 
into rule space, by computing their 6 and 5" values. Then the images of the 
"fuzzy" response patterns form a subset that swarms around the point (^r, f^) 
which corresponds uniquely to R. The swarm of mapped "fuzzy" points in the 
rule space follows approximately a multivariate normal distribution with 
centroid (9^, (Tatsuoka, 1990), which is called the bug distribution or 

state distribution associated with R. 

When two sets of "fuzzy" response patterns are mapped into the rule 
space, one can apply Bayes' decision rule for minimum errors to classify a 
student's point (dj^, f,) into one of the states. 

In summary, rule-space model is a probabilistic model for cognitive 
diagnosis and the model is applicable to any domain of interest at very low 
cost. It is a general and cost-effective method for cognitive diagnosis when 
the remediation of errors is our concern. 
Error- Diagnostic Adaptive Testing System 

Utilizing the rule-space model described above, Tatsuoka, Baillie, 
Tatsuoka (1986) have developed an adaptive testing system by which a students' 
sources of misconception (which produce bugs or erroneous rules of operation) 
can be diagnosed. A computer program that sequentially selects an optimal set 
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of items to administer to each individual, and then terminates the testing as 
soon as a specified stopping rule is satisfied, has been written on the PLATO 
system. The item-sequence selection strategy is an extension of the method 
commonly used in IRT-based adaptive testing procedures. The new system allows 
us to attain a specified level of accuracy in estimating $ and f most rapidly, 
The procedure is an application of the theory of convergence in functional 
space, the space of item response functions in this case. 

Before proceeding to a description of our diagnostic testing system, we 
must explain what 0 and ^ are. The first is the standard person parameter in 
item response theory (IRT) , commonly characterized as the latent ability of an 
individual in the domain for which he/she is being tested. However, it can 
just as well be construed as the individual's achievement level in that domain 
at the time of testing; it is this interpretation (or definition) of $ that is 
more appropriate for our purposes here. Either way, IRT postulates that the 
probability of an individual's answering a given item correctly is a joint 
function of 6 and one, two or three item parameters. Tlie latter characterize 
the discrimination power (a) , the difficulty level (b) and the "guessability" 
(c) of the item- -i.e. the chances that an examinee with absolutely no ability 
or prior achievement in the domain will answer the item correctly. The 
particular functional relation between the probability of correct response and 
5, a, b, and c may in principle be chosen at will by the researcher-- with 
some reasonable constraints such as its being a monotone increasing function 
of 6 for fixed a, b, and c. In practice, however, only two functional forms 
are widely used. These are the logistic function and the normal ogive. Here 
we use the logistic model and confine ourselves to the case of c«0, which is 
appropriate when we are dealing with open-ended (or "constructed-response" ) 
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items as against multiple-choice items. The two-parameter logistic model has 
the following form: 

Pj(Xj-lll?) - 1/[1 + exp(.1.7aj(^ - bj))] (1) 
where Xj is the binary score for item j , so that 

1 when item j is answered correctly; 
0 otherwise 

For short, the left-hand side of Equation (1) is often written as Pj (6), as 
we do below. 

We now define the true score T($) as the average of the ?j(0) over the n 

items : 

1(6) « l/n£ Pj(e) (2) 
Using this quantity as the repeated element of an n-dimensional vector, we 
obtain 

T(0 - (T(^?),T(^),.,., T(0]' 
and hence the residual (or deviation score) vector 

P(^?)-T(^?) « [Pi(^)-T(^?), P2(^?)-T(^?),...,Pn(^)-T(^)]' 
Similarly, we define the residual vector of P(^) from the binary-score vector 
x: 

P(^?)-X - [Pi(^?)-Xi, P2(^?)-X,..., Pn(0-X^]' 

We then form the scalar product of these two residual vectors, thus: 

ffj(x) « [?(e)^T(e)]*[?(e)-x] 

- E [Pj(^?)-T(^?)][Pj(^?)-Xj] (3) 
To see how it functions, we distribute the multiplication over the second 
factor to get 

f^,(x) - [P(^)-T(e)]'P(^?)-[P(^)-T((?)]'x (4) 
For fixed B, the first term is a constant. Let us see how f^Cx) varies with x 



due to the second term. Rewriting this term as 
. E [Pj(5)-T(e)]Xj 

helps us make the following observations: Without loss of generality, we may 
suppose the items to be arranged in descending order of magnitude of Pj(^). 
Then, for some m, the first m terms of the summation will have positive 
coefficients associated with the Xj , and the remaining n-m terms will have 
negative coefficients. Thus, to the extent that there is a preponderance of 
Is among the scores on the first m items (Xx,X2, . . . ,x^) and a preponderance of 
Os among the last n-m item scores, the sum (exclusive of the minus sign) will 
have a large value, and hence f^Cx) will be small. Conversely, when there are 
many Os among the earlier items, f^Cx) will be large. Since the Pj(^) values 
are in descending order of magnitude [?i(6) > ?2(^) > - > Vi(^) > Pn(^)]» ^ 
response pattern having many Is among the earlier items and many Os among the 
latter ones may be regarded as a "normal" or "typical" response pattern for 
the group in which the a^ and bj values were estimated (i.e., in which the 
items were calibrated). On the other hand, response patterns having many Os 
among the earlier items and many Is among the later items would be "atypical" 
ones. Thus typical response patterns are associated with small value fgix) 
while atypical (i.e., unusual) response patterns get larger f^ii^) values. 
Hence, fe(x) may be taken as a measure of atypicality of response patterns 
the larger feix) is, the more atypical the response pattern is. 

The function fe(x) described above suffices to serve as a measure of 
atypicality of response patterns only for the population in which the items 
were calibrated. To measure the atypicality of response patterns observed for 
examinees who are not members of the calibration population, we need to 
standardize fe(x). It was shown by Tatsuoka (1985) that the expectation and 
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variance of fg(x) for fixed 6 are 



E[fg(x)] - 0 



(5) 



and 



Var[f9(x)] - I Pj(^)Qj(0)(Pj(e) - T(9))^ 



(6) 



where Qj(^) ^ 1- ?^($) . Thus the standardized fg(x), denoted J*, is given by 



This constitutes the ordinate of rule space, the abscissa being the 
estimated value of $, Thus, each point in rule space, which represents a 
particular response pattern (or an individual who has that response pattern) 
is associated with a coordinate pair ($ , $"). The error-diagnostic adaptive 
testing procedure may be regardei as a random walk in this space, whose path 
is determined by the changing values of the vector (6, ^) as successive items 
are selectively administered to each examinee in accordance with a certain 
selection rule. 

The item-selection rule may take several forms, one of which may be 
described as follows. Suppose that, if an examinee under consideration were 
to take the entire test consisting of n items, he/she would be diagnosed as 
having a misconception that places him/her in misconception group G in 
accordance with the classification rule described in the previous section. 
Then, it stands to reason that each new item to be administered should be 
chosen in such a way that the examinee would be hurried toward his/her 
ultimate destination as rapidly as possible. This purpose will be served-- 
even though the fin?l destination is unknown- -if successive items are chosen 
so that the individual's response pattern is located as far as possible in 
rule space from its location at the time the previous item was taken. This is 
somewhat analogous to the method of steepest descent often used in certain 



f - f(tf)/(var[f(,(x)])i/2 



(7) 
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types of optimization problems. 

Once an examinee's response -pattern point stabilizes on (or converges 
to) some point in rule space, we take that to be his/her "true point", and no 
further items are administered. The next step is to determine the 
misconception group to which that point most probably belongs. This, of 
course, is a problem of statistical classification theory as described 
earlier. Thus, the problem of error diagnosis is translated into one of 
statistical classification using a suitable model. The model often chosen for 
mathematical convenience is the normal model--i.e., that 6 and f jointly 
follow a bivariate normal distribution with a specifiable centroid and 
covariance matrix specific to each misconception group. 

Since 6 and f(x) are uncorrelated, the covariance matrix becomes 
diagonal with 1/I(^) and the variance of f(x) as the diagona"* elements 
(Tatsuoka, 1985; 1990). The classification procedure is described in Tatsuoka 
& Tatsuoka (1987) . 

Method 

A Task Analysis and Cognitive Attributes 

A task analysis that was conducted in the domain of fraction addition 
problems identified 15 basic cognitive tasks for carrying out the addition of 
two fractions of which there are three types: Addition of two simple 
fractions, F + F; addition of two mixed numbers, M + M; and addition of F + M 
or M + F. These tasks are closely associated with types of items in which 
three positive integers are used to form either a mixed or fraction number. 
The 15 basic cognitive tasks are listed in Table 1. 




lb 



Insert Table 1 about here 



However, these tasks are used for generating exercise problems in 
instructional units for remediation and, further, their mastery patterns 
obtained from diagnostic classification are used for routing an examinee to 
his/her remediation unit(s) . Since these tasks are oriented toward generating 
various types of items and focused on a finer, micro -level analysis, it is 
difficult to see a close connection to the cognitive attributes discussed in 
the previous studies (Tatsuoka, 1984, 1986; Birenbaum & Shaw, 1986). In order 
to clarify the continuity of these 15 basic tasks with the attributes 
discussed in our other papers, they are grouped into eight categories: {Task 1 
or Task 2 }, {Tasks 1 and 2}, {Task 3 or 4 or 5 or 6), {Tasks 7 and/or 8}, 
{Tasks 9 and/or 10), {Tasks 11 and/or 12 and/or 13}, {Task 14} and {Task 15}. 
Let us denote these new categories as cognitive attributes A^, through Ae. 
The. eight attributes characterize 38 items with attribute involvement vectors. 
Table 2 describes the items by various combinations of attributes. The 38 
addition problems 



Insert Table 2 about here 



are also listed in Appendix I and in Appendix II with their parameter values 
in the two-parameter logistic model. 

Adaptive Testing System for Cognitive Diagnoses in Fraction Addition Problems 

An adaptive diagnostic testing system for fraction addition problems was 
developed on a computerized instructional system at the University of Illinois 

10 
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(PLATO system). The system is designed to classify examinees' responses into 
one of the predetermined misconception groups (called states of knowledge and 
capabilities). For fraction problems, the misconception groups were 
detemnined by reviewing the previous studies (Tatsuoka, 1984; Klein, 
Birenbaum, Standiford 6: Tatsuoka, 1981; Shaw & Tatsuoka, 1982). Tatsuoka 
(1984) grouped erroneous rules into several basic categories by examining 
where each erroneous rule originated. For example, if erroneous rules are due 
to lack of knowledge for making equivalent fractions, then they are labeled 
"errors in making equivalent fractions" , 

The 15 prime cognitive skills were first defined as stated earlier and 
33 knowledge states that consist of various combinations of the prime 
cognitive skills were selected based on the frequency statistics of erroneous 
rules. These 33 states constituted the "bug bank" in our adaptive cognitive- 
diagnostic testing. Paper -and- pencil tests that were given in the previous 
years were analyzed for students' errors and approximately 90 percent of the 
examinees (N-593) were classified into one of the states (Tatsuoka, 1986). 

The 10 most popular knowledge states are; 
No. 4 Cannot get the common denominator (CD) but can do simple fraction 



addition problems. 



No. 6 



Cannot get CDs for the problems involving mixed number (s) . 



No. 9 



Have problems in simplifying their answers into the simplest form. 



No. 10 



Mastery state: all cognitive attributes are correct. 



No. 11 



Can do only addition of two simple fractions (F) when they have 



the same denominators. 



No. 16 



Cannot get CDs and cannot add two reducible mixed numbers (M) . 



Also having problems with simplification of the answers. 



11 
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r 



No. 21 Non-mastery state: all attributes are wrong. 

No. 24 Cannot add a mixed number and a fraction number. Cannot get CDs. 

Don't reduce fraction parts correctly before getting the common 
denominators . 

No. 25 Cannot add the combinations of M and F numbers. Also cannot get 

CDs. 

No. 26 Don't realize that adding zero to a non-zero number a is a itself, 

a 4- 0 «= a; Identity Principle. 
Table 3 describes the ten most popular knowledge states by nine 
attribute -mastery patterns. Appendix III summarizes 33 states with respect to 
the patterns of items grouped by their underlying attribute ^^atterns , 



Insert Table 3 about here 



Knowledge state No. 10 is the mastery state; that is, all answers are 
correct while No 21 represents the state in which all answers are wrong. 
Remediation Instruction and a Routing Method based on Probabilities 

Fourteen basic skills were defined and their instructional units were 
written on the same computerized instructional system as that on which tests 
were prepared. For example, if an examinee is classified into State No, 25, 
then an automated adaptive routing system sends the examinee to the units A3,- 
- teaching what a common denominator is and how to get it- and A^, -reminding 
the student that F-f-M t3npe can be separated into 0 and the whole number part of 
the second munber and 0+d-d because students often overlook this identity 
principle. The following figure shows an example of our remediation unit. 
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Insert Figure 1 about here 



The first box shows a wrong answer by a student and starts teaching each step 
to reach the correct answer. If the student cannot get the least common 
multiple of 8 and 10 correctly, then a specific feedback based on the 
particular answer will appear on the screen. This example shows that the 
student selected a conmion denominator but did not choose the least common 
denominator. Therefore an instruction that teaches multiples of 8 and 10 
appears on the screen for the student. The box below in Figure 1 indicates 
that the student selected the right answer, 40 and got the feedback of 
"correct!". Then the computation screen goes to the next step, making 
equivalent fractions of 1/8 and 7/10 . 

The top of the remediation instruction in Figure 1 shows the routing 
index of 14 basic units. Type 11 means item type 11, which is characterized 
by the attributes A^, A2, A3 and mixed number addition problems with different 
denominators. Any examinee who is classified into one of the cognitive states 
which includes "cannot do A^" as a subset would be routed to study a series of 
instructions indexed by this label. 

However, there are several states to which the examinees diagnosed 
"cannot do A^, A2 and A3" belong. Since Boolean algebra defines a partial 
order among states derived from a given incidence matrix, the relationships 
among the cognitive states can always be expressed by a tree such as the 
example given in Figure 2. 
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Insert Figure 2 about here 



The mastery state means Group 10, all attributes are mastered. The 
states, cannot do A^" and "cannot do Ag" mean, respectively, that they cannot 
do A4 but can do the remaining attributes A^, A2, A3, A5, A^, Ay, Aq and Ag and 
that they can do A^ tTirough Ae only. 

Suppose an examinee was classified into the cognitive state "cannot do 
A3, A4, A5 and Ag" which is shown at the bottom of the tree given in Figure 2. 
The issue that arises here is that whether A3 should be remediated first and 

r 

then A5 , or A5 first and then A3. * ^ 

Tatsuoka and Tatsuoka (1987) introduced what they called i bug 

V 

distribution, which is a multivariate normal distribution with centroid (^r, 
f^) and covariance matrix » 



1/1(0) 0 
0 1 



(8) 



Bayes' decision rule to determine whether an examinee's point X in the rule 
space belongs to State A or to B is equivalent to comparing the Mahalanobis 
distance between X and A versus that between X and B. Mahalanobis distance in 
the rule space context is the same as considering the negative loglikelihood 
ratio of the two posterior probabilities of A and B given X (Lachenbruch 
1975), 

-In (Prob( A ! X )/Prob( B I X ) (9) 
By taking the position of State "cannot do A3, A4, A5 and Ag" as X and 
computing the two Mahalanobis distances, that between X and the State "cannot 
do A3, A^ and Ag", would provide a plausible rule for a computerized routing 
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system. However, in this study, the order for selecting an attribute from two 
connected states comes from the flowchart that was constructed earlier for a 
task analysis (Klein et al. , 1982; Birenbaum & Shaw, 1986). Attributes A3 
(getting the least common denominator) and A5 (reducible before getting the 
conmion denominator) can be located in the flowchart in Figure 3. 



It is now obvious that A5 should be remediated before A3. 

As for the Mahalanobis distances, that between States "cannot do A3, A^ , 
A5 and Ag" and "cannot do A^, A5, A9" is larger than that between States "cannot 
do A3, A/^, A5 and Ag" aud "A3, A^, , and Ag" . 

Computational comparison of these relationships between the order from 
the flowchart and the order based on Mahalanobis distances in rule space 
confirmed that pairs of attributes closer on the flowchart have smaller 
Mahalanobis distances than do pairs of attributes farther apart. 

The result suggests that a potentially good routing criterion for 
remediation is to compare the negative loglikelihood ratio of the posterior 
probabilities of two targeted states versus that of the state into which an 
examinee is currently classified. 

Data collection: Pretest. Remediation and Posttest in 1988 and 1989 

Three fraction diagnostic tests, pre- and post-tests and a retention 
test were given in 1988 to students in the seventh and eighth grades of a 
junior high school in a small city in Illinois. The pretest classified each 
student into one of the 33 states. Since each state is expressed as the 
mastery and non-mastery of given cognitive attributes, the examinee was 



Insert Figure 3 about here 
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assigned to the instructional units which teach the examinee his/her non- 
mastered skills. When examine.es were classified into No. 10, the mastery 
state, they did not participate in the remediation and posttest parts but they 
took the retention test three months later. 

The examinee was given a series of exercises at the end of each 
remediation unit. After correctly doing the exercises at all the remediation 
units which he/she had to complete, the examinee was given the posttest, which 
was also adaptive, ^nd a cognitive diagnosis was carried out. The posttest 
also used the same "bug bank" consisting of 33 states. 



Insert Figure 4 about here 



Three months later, the retention test was administered to the examinees 
who took the pretest and posttest. 

Figure 4 shows an example of an adaptive test that was administered to a real 
examinee, whose final classification was in State 6. The same test was given 
to the examinees three months later, and the retention of examinees' states 
was examined. 

In the next year of 1989, the same study was replicated with 191 
students . 

Results 

Table 4 summarizes the results of the six tests given in 1988 and 1989. 



Insert Table 4 about here 



16 



The two independent studies in 1988 and 1989 show a considerable 
resemblance in the classification results. In 1988, 57 examinees achieved the 
mastery state, while 39 failed in all the cognitive attributes and ended up in 
the non-mastery state (No. 21). In 1989, 34 were classified in the mastery 
state and 13 were in the non-mastery state. 

State No. 26 is the most popular knowledge state after No, 10 throughout 
the six tests. Tlie examinees in No. 26 are also very high achievers and their 
errors are "cannot do F + M type but can do all other cognitive attributes 
required in the other types of problems". Their erroneous rules are often 
"append 1 to F type and proceed with all the computations correctly" (2/5 + 4 
3/7 becomes 1 2/5 + 4 3/7) or " omitting the whole number part in the answer" 
( {0 + 4) + {2/5 + 3/7} = 29/35 ). The frequencies of such bugs were reported 
in Tatsuoka (1984) and Shaw and Tatsuoka (1982), 



Insert Table 5, 6 and 7 about here 



However, Table 4 does not show how the examinees improved after the 
remediation lessons were given. The transition of examinees' knowledge states 
before and after remediation are summarized in Table 5 for the transition 
between pre-and post-tests, in Table 6 for that between posttest and retention 
tests, and in Table 7 for that between pretest and retention test. The states 
are listed in descending order of the number of tasks mastered, starting from 
state No. 26 (13 skills mastered) down to state No. 21 (none of the tasks 
mastered) . 

Table 5 shows that twenty- five examinees moved from No. 26 to No. 10, 
(all tasks mastered). Eight from No. 25 and No. 16 moved to No. 10. Further, 
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11 from No. 6, 7 from No. 33 and No. 21 moved to Ko. 10. In the posttest, 89 
examinees (39% of the 226 students who took the posttest) were classified into 
No* 10. Fifteen examinees stayed in No. 26 on the posttest. However, four 
examinees who were in No, 26 moved back to lower-level states. In all, 16 (7% 
of N « 226) moved to lower- level states, but a majority of the examinees (93%) 
moved to higher-level states. Similar trends were found in the replication 
study in 1989. 

These changes are graphically expressed in Figure 5. The points, 21, 
16, 25, 24, 33, 6, 9, 26 and 10 are the centroids of distributions associated 
with these states. The arrows indicate that examinees' changes in their 
states between the pre-and posttests are as indicated in Table 10 and 11. The 
locations of 33 states in the rule space are listed in Appendix IV. 



Insert Figure 5 about here 



Insert Tables 8 and 9 about here 



As for changes from the posttest to the retention test, quite a few 
examinees moved back to their pretest states. Twenty-five examinees 
maintained their mastery states in the retention test, while 23 did not take 
the retention test. Forty-one examinees (48%) regressed toward lower-level 
states from their posttest state, No. 10. The examinees who were classified 
in No. 26, retained their skill level better than did those in No, 10; 43% 
stayed in either No. 10 or No. 26 while 34% moved back to lower-level states 
Overall, 48% of the examinees (N - 185) regressed toward lower- level states 
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between the posttest and retention tests. Twenty-one percent of the examinees 
maintained their posttest states, and 31% moved up to higher-level states than 
the posttest states. 

However, changes between the pretest and the retention test are 
encouraging, (see Table 7). Fifty-five examinees moved from various states to 
the mastery state, No. 10. Thirty moved to No. 26, and 25 moved to No. 6. If 
we add up these numbers, 110 out of 185 examinees (59%) were classified into 
either No. 10, 26 or 6 which are near-mastery states. However, about 6% of 
examinees regressed toward lower -level states from the pretest state to the 
retention state. 

Since quite a few examinees dropped out of our experiment before taking 
the retention test, analysis of transition states lacks in statistical power. 
Therefore, the 33 states were grouped into two categories: those with serious 
vs. non- serious error types. Of the states that have mastery tasks, more than 
eight were categorized as non-serious error states, and the remaining states 
were categorized as having serious errors. Table 10 summarizes the 
categorization of the states into which at least one examinee was classified. 



Ins rt Table 10 about here 



Nine states were categorized in the non-serious-error group while 17 were 
classified into the serious -error group. 



Insert Tables 11 and 12 



Tables 11 and 12 show 2x2 contingency tables of serious vs. non-serious 
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error groups for the pre- and post- test in the 1988 and 1989 data. Table 13 
is the corresponding table for pre- test and retention test in 1988. Tables 11 
an* 12 indicate that diagnoses using the rule-space model are very efficient. 
Indeed, we carefully designed our remediation instruction so that if cognitive 
diagnosis by the rule-space model is not correct, then remediation should not 
work well. 

Therefore we consider the results shown in Tables 11 and 12 to be a strong 
indication of the reliability of rule-space diagnoses. 

Insert Tables 13 



However, Table 13 shows that 38 examinees moved from non- serious to 
serious while 4 moved from serious to non- serious error groups. The number of 
examinees who remained in non-serious error groups was llO, v;hich is 68% of 
the examinees who participated in the retention test. 
Proportion Correct for Cognitive Attributes 

A unique feature of the rule-space model for cognitive diagnosis is that 
item-response patterns for examinees who are successfully classified can be 
converted into attribute-mastery patterns, which are usually unobservable . 
This means that one can obtain the p-values (proportions correct) for 
cognitive attributes as well as for items. If classification rates are high, 
say 90%, then the p-values can provide researchers with valuable information 
about the underlying cognitive models. 

Figure 6 shows the proportion correct scores for eight 
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Insert Figure 6 about here 



cognitive attributes, for the pretest (N 287), for the posttest (N - 225) 
and for the retention test (N = 185) in 1988. The three line graphs indicate 
that the proportions correct over eight attributes for the retention test 
maintain values higher than the pretest values and lower than the posttest 
values. Since the examinees classified in the mastery state (No. 10) in the 
pretest (N = 58) were not routed to study the remediation instruction, they 
took neither the posttest nor the retention test. Hence, the given proportion 
correct scores for 8 attributes in Figure 5 do not reflect the same total 
sample as that of the pretest, and they could be much higher than the current 
values if these mastery examinees would have studied the remediation 
instruction and took both the posttest and retention test. 
Time Required for Completing Remediation Instruction 

The average times, in minutes, for students in various states to 
complete the remediation treatment are summarized in Table 14. 



Insert Table 14 about here 



The overall average time for completing required remediation units is 37.21 
minutes across the knowledge states listed in Table 14. 49.8% of the 
examinees reached mastery or near-mastery while 5% of the examinees remained 
in the serious -error category. 
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Conclusion and SuiDmary 

Our motivation for this study was to validate the results of cognitive 
diagnoses by the rxile -space model and to demonstrate the usefulii s of 
cognitive diagnoses for instruction. This empirical validation study strongly 
indicates that the rule-space model can diagnose students' knowledge states 
effectively and remediate the students' errors quickly with minimum effort. 
The lack of knowledge in particular domains can also be diagnosed, and the 
knowledge can be supplied by instruction. 

Designing instructional units for remediation can be effectively 
navigated by the rule -space model because the determination of all the 
possible ideal item- score patterns (universal set of knowledge states) given 
an incidence matrix is based on a tree structure of cognitive attributes , 
knowledge states and items (Tatsuoka, 1990). Remediation should start at the 
states whose probability of mastery for diagnosed deficiency of skills is as 
high as possible. 
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Table 1 

Fifteen Basic Cognitive Tasks required in the Fraction Addition 
Problems 



Tasks 



Description of Cognitive Tasks 



1. 



Separate the whole number*, a from the fraction part*, b/c . 



2. 



Separate the whole number part, d from the fraction part, e/f . 



3. 



Get the common denominator CD, CD 



= c X f . 



4. 



Get the common denominator CD when c is a multiply of f . 



5. 



Get the common denominator CD when f is a multiply of c. 



7. 



Get the common denominator CD when f and c are factors of CD. 



8. 



Convert b/c before getting CD, b>c . 



9. 



Convert e/f before getting CD, e>f . 



10. Reduce b/c before getting CD. 

11. Reduce e/f before getting CD. 

12. Answer to be simplified, reduce the fraction part. 

13. Answer to be simplified, convert the fraction part to a mixed 
number . 

14. Add a whole number to the whole number part of the answer 
after converting the original fraction part. 

15. Add two numerators. 

* Item a(b/c) + d(e/f) ; F = a simple fraction, M is a mixed 

number such as 5 3/5 




Table 2 

Description of Items by Various Combinations of Attributes 



Attributes items 























1 


1 


1 


1 


1 


1 


1 


1 


1 


1 




1 


2 


3 


A 


5 


6 


7 


8 


9 


0 


1 


2 


3 


u 


5 


6 


7 


8 


9 


a ^ 0 or d ^ 0 


1 


0 


0 


1 


0 


1 


0 


0 


1 


0 


0 


1 


1 


0 


0 


0 


0 


0 


0 


a ?t 0 and d 0 


1 


0 


0 


1 


1 


1 


0 


0 


1 


0 


0 


1 


1 


0 


0 


1 


0 


0 


0 


c f 


0 


1 


0 


1 


1 


0 


0 


1 


0 


0 


1 


0 


1 


0 


1 


0 


0 


1 


0 


convert before getting CD 


1 


1 


1 


0 


1 


0 


1 


0 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


reduce before getting CD 


1 


1 


0 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


0 


0 


answer to be simplified 


1 


1 


1 


1 


1 


1 


1 


0 


1 


0 


1 


1 


0 


0 


0 


0 


0 


0 


0 


add two numeratos 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


adjust a whole number part 


1 


0 


0 


1 


1 


1 


0 


0 


1 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


Mixed numbers with c?*f 


0 


0 


0 


1 


1 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


0 


0 


0 



♦Items 20 through 38 are parallel to Items 1 through 19. 

1. a is zero if the first term is a fr.'jction. 

2. d is zero if the second term is a traction. 

3. c=f if two numbers have the same denominator. 

If e>f or /and b>f then the number (s) can be converted before addition, 

5. If (e,f) ¥■ 1 or/and (b,c) ^ 1, the numerators can be reduced before addition, 

6, Answer can be simplified by converting or /and reducing. 

7, Add two numeratos, 

8. When a mixed number answer is simplified by converting the fraction part, the whole niimber 
part must be adjusted. 
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Table 3 

Description of the top ten, the most popular Knowledge States among 
the thirty three states in the "bug list" 



Attribute Mastery pattern 



States 


1 




0 

J 




c 

J 


6 


/ 


Q 
0 




2.1 


U 


U 


U 


u 


u 


U 


U 


U 


U 


10 


1 


1 


1 


1 


1 


1 


1 


1 


1 


6 


1 


1 


1 


1 


1 


1 


1 


1 


0 


9 


1 


1 


1 


0 


1 


0 


1 


0 


1 


16 


1 


1 


0 


0 


0 


0 


1 


0 


1 


24 




-1 


0 


1 


0 


1 


1 


1 


1 


25 


0 


1 


0 


1 


1 


1 


1 


0 


1 


26 


0 


1 


1 


1 


1 


1 


1 


1 


1 


33 


1 


0 


0 


0 


0 


1 


1 


1 


1 


4 


1 


1 


0 


1 


1 


1 


1 


1 


1 


11 


0 


0 


0 


1 


0 


1 


1 


1 


1 
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Table k 

Frequenclos o£ '^Studonts Classified in Each Class 



1988 Data 1989 Data 

Groups Pretest Posttest Retention Pretest Posttest Retention 

N«287 N=226 N=185 N=191 N«175 N=171 



1 
2 
3 

5 
6 
7 
8 
9 
10 
11 
12 
13 
15 
16 
17 
18 
19 
20 
21 
23 
9A 
25 
26 



3 
2 
1 
3 
3 
26 
5 
0 
12 
57 
15 
1 
4 
0 
25 
0 
1 

u 
1 

39 
1 
9 
12 
45 



0 
0 

1 

2 
0 

25 
5 
0 

13 

89 
2 
0 
0 
0 
7 
0 
2 
3 
0 
1 
1 
1 
6 

44 



1 
0 
0 
7 
0 

25 
5 
1 
5 

55 
6 
0 
1 

2 
10 
1 
1 
0 
0 
2 

6 
11 

30 



20 
6 
0 

10 

34 

12 
1 
3 
0 

17 
0 
1 
5 
0 

13 
5 
9 
7 

27 



0 
0 
2 
2 
0 
16 
6 
1 
14 
69 
2 
0 
2 
0 
3 
1 
0 
1 
0 

1 
1 

2 
4 
33 



1 
0 
0 
7 
1 
16 
3 
0 

8 , 
51 

2 

0 

1 

0 
13 

0 

3 

0 

0 

7 

7 
10 

6 
24 

( table continues ) 
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1988 Data 



1989 Data 



Groups 


Pretest 


Post test 


Retention 


Pretest 


Posttest 


Retention 




N=287 


K=226 


N=185 


N=191 


N=175 


N=171 


27 


0 


1 


0 


0 


1 


0 


28 


1 


0 


0 


3 


0 


0 


30 


0 


1 


0 


0 


1 


0 


31 


6 


9 




3 


5 


6 


32 


2 


8 


2 


0 


5 


1 


33 


9 


k 


A 


7 


3 


4 
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Table 5 

Transition Frequencies of Students who were classified in the groups for Pre-and-Post Tests, lb. 8 Data 



Test 2#of 14 13 11 11 10 10 988887766^33300 

Tasks 



Test 1 Post 10 26 6 7 9 19 33 4 18 32 31 25 3 16 24 30 23 11 27 21 0 # 
Group 

# of Pre 

Tasks Group 

13 26 25 15 1 2 1 1 45 

11 6 11 7 4 2 1 1 26 

11 7 1 1 12 5 

10 9 3 4 2 1 2 12 

10 19 3 1 4 

9 33 7 2 9 

8 4 1 1 13 

8 18 1 1 

8 32 1 1 2 

8 31 3 1 1 1 6 

7 25 8 1 1 1 1 12 

7 3 1 1 

7 5 11 1 3 

6 16 8241111 21 1 1 2 25 

6 24 3 2 111 1 9 

5 2 11 2 

4 111 1 3 

4 12 1 1 

3 11 33122111 1 15 

( table continues ) 
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Test 2#of 14 13 11 11 10 10 988887766<»33300 

Tasks 



Test 1 Post 10 25 6 7 9 19 33 U 18 32 31 25 3 16 2k 30 23 11 27 21 0 # 

Group 



# of Pre 
TaskB Group 



3 23 1 1 

3 13 1 1 1 1 A 

3 28 1 1 

1 20 1 1 

0 21 782111 1234 ti 11 12 39 



89^4 25 5 13 3^2289617 11121 1 5 230 
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Table 6 

I 

Transition Frequencies o't Students who were classified in the groups for Post and Pretention Tests, 1908 Data ! 
. — I 

Test 3#of 14 13 11 11 10 10 9888876655^3330? i 

Tasks 

Test 2 Post 10 26 6 7 9 8 33 4 18 32 31 25 16 24 15 17 1 11 23 13 21 0 # 

Post 



# of Post 
Tasks Group 



u 


10 


25 


14 


9 4 


1 2 


1 


1 


3 


1 


1 


2 


1 


1 23 


13 


26 


12 


7 


7 


1 


1 


2 


1 








2 1 


10 


11 


6 


3 


5 


3 


2 


2 


1 


1 






1 




7 


11 


7 


1 












1 








1 


2 


10 


9 


2 


1 


1 


1 1 








1 1 






1 


4 


ID 


19 












2 












1 


9 


33 




1 


1 














1 




1 


8 


4 


1 




















1 




8 


16 








1 






1 












8 


32 


1 










1 


2 


2 








2 


8 


31 






2 




1 1 


2 




1 








2 


7 


25 








2 




1 












3 


7 


3 




1 






















6 


16 






1 


1 






1 






1 1 




2 


6 


24 
















1 












30 




















1 






3 


23 








1 


















3 


11 








1 














1 




3 


27 










1 

















( table continu»s ) 
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Test 3#of 1^1 13 11 11 10 10 986887665- 54 33 3 0? 
Tasks 

Test 2 Post 10 26 6 7 9 8 33 A 18 32 31 25 16 2A 15 17 1 11 23 13 21 0 # 
Post 



# of Post 
Tasks Group 



0 21 11 

? 0 10 1 2 3 1 1 62 



55 30 25 5 5 1 A 7 1 2 A 11 10 6 2 1 1 6 6 1 2 102.287 
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TrMisition Frequencies of Students who were classified in the groups for Pre-and Retention Tests, 1988 Data 



Test 3 


# of 


1^ 


13 


11 11 10 


10 9 


8 8 


8 


8 7 


6 


6 


5 5 


A 3 3 3 0 0 




Tasks 
























Test 1 


Post 


10 


26 


6 7 9 


8 33 


^ 18 


32 


31 25 


1^ 


24 


15 17 


1 11 23 13 21 0 




Post 


























Group 

























# of Pre 
Tasks Group 



u 


10 


10 


1 


2 


3 


1 


1 






39 


57 


13 


25 


16 


12 


2 


2 




1 






12 


45 


11 


6 


6 


3 


6 




12 1 


1 






6 


26 


11 


7 




1 


1 






1 1 






1 


5 


10 


9 


4 


1 


1 












6 


12 


10 


19 


1 


1 




1 






1 






4 


9 


33 


5 




2 












2 


9 


6 


4 










1 


1 






1 


3 


6 


18 














1 






1 


8 


32 


















2 


2 


6 


31 


2 


1 








1 






2 


6 


7 


25 


1 


1 


1 


1 


1 






1 


6 


12 


7 


3 


1 


















1 


7 


5 








1 


1 






1 




3 


6 


16 


3 


2 


1 


1 


1 1 


12 1 1 


2 




9 


25 


6 


24 




1 


1 




1 


1 




1 


4 


9 


5 


2 










1 


1 








2 


4 


1 




1 








1 






1 


3 



( table continuf ) 



Test 3 # of 



lA 13 11 11 10 10 9 8 8 8 8 7 6 6 5 5 A 3 3 3 0 0 



Tasks 

Test 1 Post 
Post 
Group 



10 26 



8 33 A 18 32 31 25 16 2A 15 17 1 11 23 13 21 



# of Pre 
Tasks Group 



12 
11 
23 
13 
28 
20 
21 



1 1 

2 15 

1 
4 
1 
1 

8 39 



55 30 25 5 5 1 4 7 1 2 4 11 10 6 2 1 1 6 6 1 2 102 287 



*39 studrnts who took Test 1 and wre classified as Group 10 did not take Test 3. 



ERIC 



4i 



Table 8 

Percentage of Transition Frequencies of Students Who are Classified 
in States 21 . 16 , 24 . 25 , 33 , 6 , 9 . 26 . & 10 . 



Posttest states 



pretest 






















other 




states 


10 


26 


6 


9 


33 


z5 


16 


24 


11 




states 




10 


























26 


57 


33 


2 


4 


0 


0 


0 


0 


0 


0 


4 


100 


6 


42 


27 


15 


8 


4 


0 


0 


0 


0 


0 


4 


100 


9 


25 


33 


17 


17 


0 


0 


0 


0 


0 


0 


8 


100 


33 


78 


0 


22 


0 


0 


0 


0 


0 


0 


0 


0 


100 


25 


68 


8 


8 


0 


0 


8 


0 


0 


0 


0 


8 


100 


16 


32 


8 


16 


4 


4 


0 


4 


0 


0 


0 


32 


100 


24 


33 


0 


22 


0 


11 


0 


11 


0 


0 


0 


23 


100 


11 


20 


20 


7 


13 


0 


0 


0 


7 


0 


0 


33 


100 


21 


18 


21 


5 


3 


0 


0 


10 


0 


3 


3 


37 


100 



Table 9 



Percentage of Transition Frequencies of Students Who are Classified 
in States 21. 16. 24. 25. 33. 6. 9. 26, & 10. 









Retention 


test 


states 








pretest 






















did not 


states 


10 


26 


6 


9 


33 


25 


16 


24 


11 


21 


take test 


10 


18 


2 


4 


5 


2 


0 


0 


0 


0 


0 


68* 


26 


36 


27 


4 


4 


0 


0 


2 


0 


0 


0 


27 


6 


23 


12 


23 


0 


8 


0 


0 


0 


0 


0 


23 


9 


33 


8 


8 


0 


0 


0 


0 


0 


0 


0 


50 


33 


56 


0 


22 


0 


0 


0 


0 


0 


0 


0 


22 


25 


8 


8 


8 


0 


0 


8 


0 


0 


0 


0 


50 


16 


12 


8 


4 


0 


0 


4 


8 


4 


8 


0 


36 


24 


0 


1 


1 


0 


0 


1 


0 


0 


0 


0 


44 


11 


13 


13 


13 


0 


0 


7 


13 


7 


0 


0 


13 


21 


5 


8 


13 


0 


0 


13 


10 


3 


5 


5 


21 



Note . * 39 out of 57 examinees who took the pretest and were 
classified as No. 10 state did not take the retention 
test. (287 examinees at P.retest but 102 dropped out of 
this study without taking the retention test) 



Table 10 



Two Categories of Error Types: Serious vs. Nonserious 



Serious Errors Nonserious Errors 

Classes # of mastered tasks Classes # of mastered tasks 



1 
i 


A 


o 
0 


1 


26 


1 0 
li 


0 


1 c? 
io 


Q 

0 




/: 
D 


i i 


o 




/ 


0 


/ 


i i 


/. 




7 


I. 
t+ 


0 


iU 


Q 


ZD 


7 


c 
D 


o 
0 


iU 


C 

D 




/: 
D 


c 
D 




1 CS 

iU 


7 


J. D 


D 


7 




Q 

y 


8 


2 


5 


8 


31 


8 


9 


30 


4 


9 


32 


8 


10 


17 


4 








11 


1 


4 








12 


27 


3 








13 


23 


3 








14 


28 


3 








15 


11 


3 








16 


13 


3 








17 


21 


0 
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Table 11 

2x2 Contingency Table of Seriousness of Error Classes and 
Pre-Posttests. 1988 Data 



Posttest 



Pretest 


Serious 


Nonserious 


Total 


Serious 


21 


93 


114 (51%) 


Nonserious 


1 


107 


108 (49%) 


Total 


22 


200 


222 




(10%) 


(90%) 





45 



Table 12 

2x2 Contingency Table of Seriousness of Error Classes and 
Pre-Posttests . 1989 Data 



Posttest 



Pretest 


Serious 


Nonserious 


Total 


Serious 


19 


75 


94 (55%) 


Nonserious 


1 


77 


78 (45%) 


Total 


20 


152 


172 




(12%) 


(88%) 
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Table 13 

2x2 Contingency Table of Seriousness of Error Classes and 
Post-Retention Tests. 1988 Data 



Posttest 



Pretest 


Serious 


Nonserious 


Total 


Serious 


15 


4 


19 (11%) 


Nonserious 


38 


110 


148 (89%) 


Total 


53 


114 


167 




(32%) 


(68%) 





47 



Table 14 



Time required for completing the remediatioii instructional units. 





1*1 niA 1 n 

X jLUIC XJLi 




2 


70 


3 


s 




0 


\j 




0 


7 


12 


0 


9 


1 


2 


11 


7S 


8 


13 


79 


7 


16 


82 


1 


18 


33 


0 


19 


21 


0 


21 


116 


4 


25 


32 


5 


26 


2 


9 


31 
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Appendix L 
38 fraction addition problems 

I. 2U3^= 17. U|= 33. + m = 

6 6 4 4 36 35 

^* 5^ 8 " 15 10 ^ - 3 9 

5 5 ^ 5 5 7 7 

4. 2l + 4= 20. 3^ + 4= 36. ^ + 1 = 

2 4 4 4 5 5 

5 l+iiO= 21 2.+ li.= 37 1+1 = 

^•2 7 7 12 6 8 

6. 35- + 4|-= 22. i- + y-= 38. ^ + J = 

7 7' 7 7 7 7 

7. 1 + 2-= 23. lj- + 2| = 

8. 24. l + 2f = 

:0. 26. = 

II. l.i= 27. 1.1 = 

12. 2|+ll= 28. l|+l| = 

13. 3l + 2|= . 29. 4 + 1 = 

M= 31. 3i+ii = 

16. 1^ + 1= 32. 2j-+3| = 
5 5 0 0 
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Appendix II 
Values of Item Parameters (N-595) 

item a values b values 

1 .5961 -.6818 

2 1.3826 .3310 

3 .9763 -.6910 
A 2.0791 .1002 

5 1.80A3 .^1^95 

6 .7118 -.7733 

7 1.2399 -.5675 

8 2.5A06 -.0^20 

9 .7030 -.A912 

10 1.299A -1.1A18 

11 2.A550 .0A77 

12 .8223 -.9559 

13 2.7216 .235A 
lA .8753 -.9561 

15 3.A782 .0881 

16 1.3121 -.8332 

17 1.12A8 -1.2200 

18 2.235A .2188 

19 1.1463 -.9001 

20 1.003A -.502A 

21 1.8327 .8211 

22 1.1226 -.5687 

23 2.6170 .1793 
2A 2.7318 .4305 

25 1.1263 -.4051 

26 1.2935 -.6279 

27 5.5522 .1568 

28 1.1126 -.4178 



( appendix continues ) 
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item 


a values 


b values 


29 




"1.0376 


30 


3.2306 


.1782 


31 


1.1067 


-.6542 


32 


3.4223 


.3703 


33 


.8436 


-.5876 


21* 


4.1896 


.1788 


35 


1.1425 


-.6935 


36 


1.1988 


-.9^17 


37 


4 .1909 


.2970 


38 


.8675 


-.8572 
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Apptndix IXX 

Description of States by Basic Types Riven in Appendix IV 





States 


















1 


X 


Bo 


B3 


Be 














2 


B4 


B5 


B7 


^8 














3 


Bi 

X 


Bo 


B3 


^6 


B9 


BlO 


^13 








4 


Bi 


Bp 


B3 


B4 


B5 


Be 


B7 


B8 






5 


Bi 


Bo 
z 


B3 


Be 


Bll 


B12 


Bl4 








6 


Bi 


Bp 


B3 




B5 


Be 


B7 


B8 


B9 


BlO 


7 


Bi 


Bp 


B3 


B4 




Be 


B7 


B8 


Bll 


Bl4 


8 


Ba 


B5 


By 


B8 


B9 


Bio 


Bll 


B12 


Bl3 


Bl4 


9 


Bi 


B2 


B3 


Be 


B9 


Bio 


Bll 


B12 


Bl3 


Bl4 


10 


Bi 


B2 


B3 


B4 




Be 


B7 


B8 


Bg 


BlO 


11 


Bl 


B2 


B3 
















12 


Bl 


B2 


B3 


H 














13 


Ba 

H 




















14 


B1I 
1 1 


Bl4 


















15 


Bl 


^2 


B3 


B9 


Bio 












16 


Bl 


B2 


B3 


B4 




B7 










17 


Bl 


B2 


B3 


^1 


^14 












18 


Bl 


B2 


B3 


B4 


B5 


B7 


B9 


BlO 






19 


^1 


'^2 


B3 


B4 


^5 


B7 


B9 


BlO 


Bll 


Bl4 


20 


B9 




















21 






















22 


Bl 




















23 


Bl 


B2 


B3 
















24 


Bl 


B2 


B3 


B4 




B7 










25 


Bl 


B2 


B3 


B4 


Be 


B7 


Be 








26 


Bl 


B2 


B3 


Ba 


Bs 


B7 


Be 


B9 


2 10 


Bll 


27 


Bl 


Bg 


Bio 

















( appendix continues ) 



states 



28 


Bl 


Bs 


Bio 










29 


Bl 


B2 


B9 


BlO 








30 


Bl 


B2 


B3 


Bg 








31 


Bl 


B2 


B3 


B« 


B5 


B7 


Bg 


32 


Bl 


B2 


B3 


B« 


B5 


Be 


By Bg 


33 


Bl 


B2 


B3 


B4 


B5 


Be 


By Bg 



APPENDIX IV 
Location of Groups in Rule Space, {$. C) 

Group 6 C Standzrd Error 

o£ Estimates 

1 -.666 -.738 .210 

2 -.98A .906 .233 

3 .2U5 1.953 .097 

4 --.168 -3.0^2 .137 

5 -.020 2.876 .112 

6 .'421 -.296 .11^ 

7 .150 -.028 .096 

8 .A67 5.22X .1?2 

9 .580 2.1A7 .16^ 

10 1.818 .507 .778 

11 I.IBU .1G5 .2^9 

12 -.833 -.668 .27.3 

13 -1.3A3 .949 .267 
lA -.877 6.11? .226 

15 -.042 4.574 .115 

16 -.639 -.333 .208 

17 -.353 4,430 .171 

18 .078 1.73*-- .100 

19 .250 1.306 .097 

20 -1.316 3.670 .264 

21 -2.979 .597 .863 

22 -1.902 .709 .373 

23 -.802 -.327 .221 

24 -.292 -1.890 .160 

25 -.227 -2.354 .148 

26 .848 .378 .221 

27 -.483 6.187 .191 
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( appondix continues ) 



Group B r Standard Error 

of Estitnat.es 

28 -.9A7 2.858 .231 

29 .05A 5.A28 .102 

30 .075 2.332 .100 

31 .277 .088 .099 

32 .213 -.669 .096 

33 .305 -1.022 .101 
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Figure Captions 

Figure 1 Examples of remediation instruction prepared on PLATO 
system. 

Fi gure 2 . A tree representation of nine cognitive states. 
Figure 3 . A flowchart for solving fraction addition problems. 
Figure 4. An example of a student's performance (Student No, 48). 
Figure 5 ._ A map showing transition of states in 1988 data. 
Figure 6. Percentage of correct scores for eight attributes 
described in Table 2. 
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Quest i on type 1 1 o f 1 4 * 

3-|- + 4-^= 7 8/40 ^^"^'"^'"^ answer is 
8 i ^ not correct « 

Let's do this problem step -by -step. 

First, add the whole number parts: 3 + 4 = 7 ok 

fire the two denominators the same? n 
.The deTiominators are unequal. They are 8 and IB'. 

Choose the least common denominator of 8 and- IS': }> o£i 
8.0' is a common denominator, but let's use 4.0' 
because it's the LERST commo7i denominator. 
The multiples of 8 are: 8 16 24 32 4.©" 48 56 64 72 8S' 
The m-altiples of 1£ are: 2.8* S.Q' 4.0" 5.ef 6,0" IfS 8.9* 
For the denominator choose the SMRLLEST numiber 
that is a multiple of BOTH 8 and 1.9'. 



Question type 11 of 14- 

li? nc/t correct. 

Let's do this problem step-by-step. 

First 5 add the whole number parts: 3 + 4 = 7 ok 

fire the two denominators the same? n 

The denominators are unequal . They are 8 and IB'. 

Choose the least common denominator of 8 and 12: 4.9' 
Correct ! 

Now 1 et ' s make equ i va 1 ent f ract i ons : 
6 40 10 4i3 



Figure 1 Examples of remediation instruction prepared on PLATO system 
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Figure 2 A tree representation of nine cognitive states 
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INPUT 



F + F ? 



no 



yes 



s i rnp 1 i f y 



f ? 



no 



yes 
^ 



add 
numerators 



can answer be 
simpl i f ied ? 



no 



yes 



^ F+M^M+F^M+M 



method fi 



method D 



ac-fb ^ fd+e 



common 
denominators 



II III 



IV 



equ i va 1 ent 
f r act i ons 



done 



s i mp 1 i f'y 



a+d 



F = fraction (b/c) 
M = mixed number (a b/c) 
a,d = 1st, 2nd whole numbers 
b^e = 1st, 2nd numerators 
c,f - 1st, 2nd denominatoT^s 
Method fl: convert improper fraction 
Method B: separate fraction and whole number part 



I: LCD prime factoring 
II: mu 1 1 i p 1 es mc. = n f 
III: one denom = multiple 

of another 
IV: automatic cf 



Figure 3 A flow chart for solving fraction addition problems 
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student 48: 



17, 


items 


done 


, total 


score = 

At. 


djl = 
1) 


F. 1537, 


grpi 
3/ 


= 
5 


6 
+ 


5 


dj 2-= 
1/ 5 


2) 


5 


B 


1/ 


.-v 
iL 


+ 


1 


IB/ 7 


3) 


25 


1 3 


4/ 


5 


+ 


5 


3/ 5 


4) 


27 


1 


1/ 


5 


+ 




!/• 4 


5) 


29 


1 


4/ 


7 


+ 




1/ 7 


6) 


32 


JEf 2 


1/ 


8 


+ 


3 


5/ 6 


7) 


21 


1 


2/ 


7 


+ 




18/-1 2 


8) 


24 


0 


1/ 


5 


+ 


2 


5/ 3 


9) 


2 


1 


2/ 


5 


+ 




12/ 8 


1.0-) 


1 8 


1 


4/1 


5 


+ 




1/1 B' 


.11) 


34 


1 


1/ 




+ 




4/ 9 


12) 


13 


SS 3 


1/ 


6 


+ 


2 


3/ 4 


13) 


37 


1 


5/ 


6 


+ 




1/ 8 


14) 


15 


1 


1/ 


2 


+ 




3/ 8 


15) 


8 


1 


1/ 


3 


+ 




1/ 2 


16) 


7 


1 


3/ 


5 


+ 




7/ 5 


17) 


9 


i 1 


4/' 


7 


+ 


1 


12/ 7 



/group 2 " 
13 

2.5J0'58, grp2==31 



* Mahalanobis distance between the student's position and the centroid of 

State 6 in Rule 'Space. 
**Mahalanobi s distance between the student's position and the centroid of 

State 31 which is the second closest. 
***Items given . 
+Itemscores 



Figure 4 An example of a student's performance (Student No. 48) 
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Low Ability 



Unusualness of Response Patterns 



Unusual Response 





Pretest Posttest 



Ability 
level 



Higli Ability 



Figure 5. A map showing transition of states in 1 988 data 
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Or. Fno DrMjow 

UfoCMtUM Cdtitr 
Ai<u*4n*. VA 22314 

(4 CofM.) 

f <rM«td DrcMtMw IUmici^ 
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Rjv«rpl*« S*ii* 405 
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Dr. RjcUrJ D«ri« 
Cr«4«tu S^««J of E4itc4Uo« 
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Dr. NtMT Ei4r«4|< 
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Dr. J«k* Qlu 
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Dr. LomiM D. C)4« 
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Dr. Kkh»e4 I- Ftrr»»«« 
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Dr. Ocf^iH FM<i«r 
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Dr. MyrM r**cil 
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Dr. AUrW R- F«|»y 
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Df. R*^ D. O^VUm 

HFI»»A, ViC*l3 
912 So«t& W#o4 SttTd 
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Dr.JMktOiiSoH 
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Dr. H<«ry R Hii«r 
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Dr. D<N7* Hiriu^ 
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Dr. TWMiM M. Hif*<l 
ACT 

p. a Box I6» 
U«« Oty. lA )Z20 
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Dr. E*(1 H«it 
Uxvrrvly •( W*tkia|tM 

Dr. H«y»k H«y«k 

Dt. M»mt J. Jft<i 
C4«ut lor t*4 $<»*y *< 

Ln«<« Utivcnily 
P. a Boj 9)» 
2X0 RJD Ui4<« 
THEHETKEJOAKDJ 

Df. R<^n J»k»«ro*4 

Dr. Xmmtt Jo* J 4" 

D«^rt»«ttt o< 5uMt>a 
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Df. f <*r( iokuo* 

AJ»^MiSM,NH 171)1 
fw C f^cw Da«|Ui H. JoM 
Rai|<n. TVf Suit VtrMttty 

Dt. J«U Jo«i4«i 

Ua<vvT«if)r c( MMikipt 
A*« A^Wf. MI UlCM 

Df. hcuA J«aL«T 

DwfartMat SaMUs 
PuuiAf^ TA 

D«. MifMl J-wt 
CtTMp* M<llo« Usnrfwy 
D>rf«n»<t( o< Py A ttop 
Scxaky PiM 
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Df. J. L )Cai*i 
0>4< 442/JlC 

Havii 0£«t SytU«i Cc«Ur 
Di«|O.CA 9n»XI00 



Dr. UKk*4l lCAfU4 

sen £iM«Ww«r Av<tM 
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101 tUIil Hd! 

725 WH|k( Si. 
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SOUTH KOREA 
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Dt. WUkia Kac^ 
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Utiwrticy T<»Ai AwJi* 
AwmTX 71703 

Dr. Kc*kU K»i«vti7 

CitMck Mcik* Uiivtnity 

rS(Wkw|*L, PA 1)713 

Dr. RicUf^ J. Kj>*Uk 
SOod o< U4wtrul 

OruM« Hall 
far^M Uaivtfuf7 
WutUUy«tu,IN 47907 

Dr. Jiao Kma 
C»a*^Uf.>M»4 S4»ca(io« 

UMVtnity •< IIUmi 

Dr. PattkJi K;1V>*f • 
AFHRlTMOa. 
i(*«AiAF 0.7X7123} 

Mk Ca««^ UiKy 

Sf<MfMU«, MD X«64 

Dr. Matty L^tum*t 
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210O S«oft4 St^ SW 
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Dr. MktMl LrvfM 

210 Bimlfea 
13tOS««ASatkS<rr«t 
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Dr. a«it«t Lt«ti 
BiMMiMHl T«»i>a| ScrviM 
PriMMM, KJ 0*54}^ 
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D«f«<tM»t •< SUlUlHt 
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Dr. UA^a C LiM 
OrWMW ScW»4 
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Dr, R»Wn L U«* 

ftMWfr, 00 IQ>0Ma4* 

Im. (Aoae UW» r) 
T»«<K*J i*4Tf»tttt| Sytuiu 
Dfwi*< 

r;a»a«u>M 
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PiU. Drr»4 F. U4wm 
C««<|a of B4»c*t>«« 

Otj, lA 

Dr. Rx&aH L*rat 
ACT 
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I<Hn Cit7. lA )220 

Dr. 0#of|« a. M»crc»4y 
Dcp«n»<it 0* MtuafTBUt 
SuAaUc* A B^Uauoa 

UiiwmiTjr •( MwyUaJ 
CXUj* P,rt.M0 2070 

V«r> M. M«t« 

NPRDCCoJ* WI 
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Dr. B»>M Maa4«t 
0«tffi« Mt*4« Uai>rrnif)r 
4«0Uaiv«n<t7 D<-iv4 
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Dr. U»4n P. MirtkaU 
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Dr. EliuWni Marti< 
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Dc N*«M« Martu 
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ACT 

f . a Ao« i6t 

TnKttom^ HI OtMl 
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Dr. W. Aiti NKo*«»4«f 
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&r*«t»l AmmUsI Ur lt<t<»rtk 
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Dr. J»^Onuti 

Mul $4*f 2>»-l 

HA5A Am R*»«Jrcft Ccaur 

Dr. Bwmft f iiMf 

KAM A»« fUM*K* C«*i<r 
M«fl<« PkM. CA MD) 

Df. Nt«r ^. fMikr 
B4»atiMAl Tt»4M| S«rvw« 

OBX> Tmcui (ffviM, S«i« 30 
Om DwfMl CftK KW 
WMkMilM, DC aoQM 

Dr. R*7f» 

inO Mifk AxiM 
B*»»it««. IL (0201 

n. ftUmtUn 

Rm Fria T»«uBiBt 47 
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BELGIUM 

Dr. Kty S> f «m 

AWCfERin) 
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C" (MD) Dr. A*Uii« Nri 
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M>Ht<n UXXO. rs*! 
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Dtfi. «< AiaUUtntiM ichtcAt 

Dr. f (Ur riKlIt 
B<rfc<k7.CA »(720 



Dr. MmU T^Uc* 
Dr^iraBcai «< riTcko^QT 
Uttvtnkjr •< C««orWo 
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Dr. rittr r*b*« 
Utivrritiy C*4*<i4* 
D<f«rtr::<t( M N><i*l«0 
•m)4c7, CO KUO^OyM 

Dr. Jm4^ f Mtkj 
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A»«Twa« ftf^^pcA\ Am«c 
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Dr. Mkft D. IU«4am 

ACT 
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(*«t Oty, lA SZ24J 

Dc 4. W«ky Brp/.i 
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PriMMM.Rr OU42 

Dr. Usm RctaK4 
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fA iUM 

Dr.OiJVrt B>-»t4 

»«Uf«t.<sKY 11714 

Mr. W. A. Ru»« 
Hof4, Hta«a Ficc»n Dtvuioa 
NamI TaiBiii Sx»m»j Otiut 
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lOV) RrvcBKA Pirk>«y 

Dr. Ufti* O. Ro^^rti 
Sor»««, BiMOOA. ^4 
TnH]i«<\ibo« Pr9|rtM 

Wuki«tt>«.DC 2»10 

Mr. Lmu R««woi 

Df^crOMit o< iutMoei 

1« tUu>H>fl 

72} Wo^i U. 

Dr. Dout^ RbW« 
Ju(utk« DfftriMoC 
i<tt%<4 Oiicr, Roo«i «0t 
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C«*W>4r, MA 021 M 

Dr. FmiIo StarjMu 
DTf«rt»<a{ W fiyUo^x? 
Uamrtftj «<T<imw< 
)lOt AmU* P<>r 8i4|. 



Dr. Wiltrr S<^»rt4<r 
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KcUMrjk Pa 1)2«) 

Dr.Mt/ySaraQ 
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Oifck^i. CA 92001 

Dc Myru F. Sahara 
Dir*«i»< 
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1200 Wot TiWrKoai 
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Dr. Vtknc L UtU* 
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Mr. fU^rd i.'Skmltcs 
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Mi. ICitkkci S4««ktt 
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Df. R>a4*ll Sk»«il«r 
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Wi(bi«|to«. DC yan Kto 

Dr. ZiU M. SiaiUJ 
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RcKtrck Lsbor>v»fy 
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Dr. l>f«k St*t»ii» 
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Dr. R«*<ft SaillK 
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Dr. Ricktrd E. S»<rm 
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ACT 
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Dr. Mtrtkt Uoctjni 

Dr. WiiU« S4o«( 
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Dr. TauMlA 
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Mt.l S4«f (U-T 

Dr. Div\4 TkuMt 
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Mr. TVoM«t I. 
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Dr. OftryT>o«tM«i 
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U« DkrKCAf^lSZ-^tOO 

Dr. JcrrjrVoit 
D<f«n»<«t *f TiytVotofy 
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Dr. Jac^tn VMc<k( 
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Dr. TkMM A. Wtm 
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Dr. D»v« J. W«i« 
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